Hi, @Aneurysm9 @rapphil as I said here
#13849 (comment)

So after chatting with @gouthamve on Promcon, he suggested I should just choose a simple option for implementing support for sending the metadata in the remote write exporter and put it behind a config option. This is what I came up with. I would like some feedback on this before I add tests. Basically the same way we batch the metrics into a couple of requests that are below the max size of a single one, we repeat the same for metadata and send it as a separate set of requests.

The one thing that stands out is that metadata is listed as Future Plans on the remote write spec.

We talked about this in the Collector Sig on 10/18 and @jmichalek132 provided some clarification that upstream Prometheus already sends metadata but just less frequently. Current proposed implementation here would send it more often so that we don't have to maintain a metric -> metadata map.

Also, implementation is opt in and behind feature gate here.

Also, PRW 1.1 spec is in flight which contains metadata related changes. I have not read it yet and thus cannot provide a summary.

Please provide corrections for anything I may have missed here.

The one thing that stands out is that metadata is listed as Future Plans on the remote write spec.

We talked about this in the Collector Sig on 10/18 and @jmichalek132 provided some clarification that upstream Prometheus already sends metadata but just less frequently. Current proposed implementation here would send it more often so that we don't have to maintain a metric -> metadata map.

Also, implementation is opt in and behind feature gate here.

Also, PRW 1.1 spec is in flight which contains metadata related changes. I have not read it yet and thus cannot provide a summary.

Please provide corrections for anything I may have missed here.

• Metadata is currently send by default already in prometheus when using remote write (config).
• For the PRW 1.1. spec, the aim is to basically do the same thing as OTLP already does, which is sending the metadata in every request with the metric, I would be also willing to implement these changes in remote write exporter.

Pipeline failiures are unrelated to my changes, and come from other exporters.

Re-tested in our dev env, to make sure everything still works (sending data to mimir).

Metadata nicely showing up in Grafana.

We do see a roughly doubling of the requests sent (which makes sense, for every timeseries request we send and extra request carrying metadata request. But it is disabled by default. Also in the coming remote write 1.1 metadata will be send along side timeseries data in the same request anyway (this is also done in OTLP).

No errors in logs of either mimir / otel collector or in metrics.

Hi, @rapphil @dashpole could you please review this again when you have a chance so we can get this merged :) ?